Skip to main content

Full text of "linux.conf.au 2016 - Internet Archive: Universal Access. Open APIs"

See other formats


INTERNET 



ARCHIVE 




Internet Archive: 
Universal Access. 
Open APIs. 



linux.conf.au 2016 
Open Knowledge Mini-Conf 





VM Brasseur 



Ica201 6@vmbrasseur.com 

@vmbrasseur 
IRC: vmbrasseur 
http://vmbrasseur.com 




Hewlett Packard 

Enterprise 



Alexis Rossi 

Director of Media & Access 
Internet Archive 
alexis@archive.org 
@alexisrossi 
http://archive.org 



E— ARCHIVE 





Vicky Brasseur 



Ica201 6@vmbrasseur.com 

@vmbrasseur 
IRC: vmbrasseur 
http://vmbrasseur.com 




Hewlett Packard 

Enterprise 



Alexis Rossi 

Director of Media & Access 
Internet Archive 
alexis@archive.org 
@alexisrossi 
http://archive.org 



E— ARCHIVE 





Rex Feature 





Some Handy Links 



https://github.com/internetarchive 

https://archive.org/details/lca 2016 -ia-apis 

https ://www.zotero . org/g rou ps/i nter net_arch i ve_- 
_open_apis_and_examples/items 




A Tour of Internet 

Archive 




Internet Archive 




501(c)(3) non-profit, founded in 1996 
Registered library in California, USA 
Partnerships with hundreds of libraries, 
museums & universities 
24 petabytes of unique data 



Where does the money come from? 

• 40% digitization projects 

• 20% web archiving projects 

• 40% foundations & donations 




4UM 



ERNET ARCH.VE 








Photo: Jason Scott 




Archived web sites 

• 400+ billion web captures, 1996 to present 

• Wayback Machine updated within hours 

• 700,000 people per day 

• 80TB crawl open for bulk download 



6 

13 

20 

27 

5 

12 

19 

26 

1 

6 

15 

22 

29 

L _ 



INTERNET ARCHIVE 




http://www.gnu.org/ 



BROWSE HISTORY 



http://www.gnu.org / 

Saved 4,872 times between January 26, 1998 and June 16, 2014. 

PLEASE DONATE TODAY. Your generosity preserves knowledge for future generations. Thank you. 



■u 



ui. 




1997 1996 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 



2014 





JAN 














FEB 














MAR 














APR 










1 


2 


3 


4 














1 














1 






1 


2 


3 


4 


5 


7 


8 


9 


10 


11 


2 


3 


4 


5 


6 


7 


8 


2 


3 


4 


5 


6 


7 


8 


6 


7 


8 


9 


10 


11 


12 


14 


15 


16 


17 


18 


9 


10 


11 


12 


13 


14 


IS 


9 


10 


11 


12 


13 


14 


15 


13 


14 


IS 


16 


17 


18 


19 


21 


22 


23 


24 


25 


16 


17 


18 


19 


20 


21 


22 


16 


17 


18 


19 


20 


21 


22 


20 


21 


22 


23 


24 


25 


26 


28 


29 


30 


31 




23 


24 


25 


26 


27 


28 




23 


24 


25 


26 


27 


28 


29 


27 


28 


29 


30 










MAY 














JUN 








30 


31 




JUL 














AUG 












1 


2 


3 


1 


2 


3 


4 


5 


6 


7 






1 


2 


3 


4 


5 












1 


2 


6 


7 


8 


9 


10 


8 


9 


10 


11 


12 


13 


14 


6 


7 


8 


9 


10 


11 


12 


3 


4 


5 


6 


7 


8 


9 


13 


14 


15 


16 


17 


IS 


16 


17 


18 


19 


20 


21 


13 


14 


15 


16 


17 


18 


19 


10 


11 


12 


13 


14 


15 


16 


20 


21 


22 


23 


24 


22 


23 


24 


25 


26 


27 


28 


20 


21 


22 


23 


24 


25 


26 


17 


18 


19 


20 


21 


22 


23 


27 


28 


29 


30 


31 


29 


30 












27 


28 


29 


30 


31 






24 


25 


26 


27 


28 


29 


30 




SEP 














OCT 














NOV 








31 






DEC 








2 


3 


4 


5 


6 








1 


2 


3 


4 














1 




1 


2 


3 


4 


5 


6 


9 


10 


11 


12 


13 


5 


6 


7 


8 


9 


10 


11 


2 


3 


4 


5 


6 


7 


8 


7 


8 


9 


10 


11 


12 


13 


16 


17 


18 


19 


20 


12 


13 


14 


15 


16 


17 


18 


9 


10 


11 


12 


13 


14 


15 


14 


15 


16 


17 


18 


19 


20 


23 


24 


25 


26 


27 


19 


20 


21 


22 


23 


24 


25 


16 


17 


18 


19 


20 


21 


22 


21 


22 


23 


24 


25 


26 


27 


30 










26 


27 


28 


29 


30 


31 




23 


24 


25 


26 


27 


28 


29 


28 


29 


30 


31 









30 




INTERNET ARCHIVE 

http://www.gnu.org/ BROWSE HISTORY 

http://www.gnu.org / 

Saved 4,872 times between January 26, 1998 and June 16, 2014. 

PLEASE DONATE TODAY. Your generosity preserves knowledge for future generations. Thank you. 

. .ki j.uA. , 

1996 1997 1996 1999 2000 2001 2 





JAN FEB MAR APR 

1234 1 1 12345 



56789 10 11 2345676 2345678 6789 10 11 12 

12 13 14 15 16 17 18 9 10 11 12 13 14 15 9 10 11 12 13 14 15 13 14 15 16 17 18 19 

19 20 21 22 23 24 25 16 17 18 19 20 21 22 16 17 18 19 20 21 22 20 21 22 23 24 25 26 

How do we decide what to archive? 

• People pay us 

• Organizations donate crawled content 

• We crawl on our own behalf 

• Deep crawl on popular sites 

• Broad but shallow crawl on known domains 

• Targeted crawls 



H 



VMBrasseur 



INTERNET ARCHIVE 




BROWSE HISTORY 



432 billion web pages saved over time, donate 



.I**'* 



B 



y—v 



I ll\ 









"&■ ■ 2 
U L 




0 



Tools 




Subscription Service 



Save Page Now 



Wayback Machine Availability API 


Archive-lt enables you to capture, marl 


Eje and 


http:// 


SAVE PAGE ■ 


Build your own tools. 


search collections of digital content wil 


but any 







Word Press Broken Link Checker 

Banish broken links from your blog. 

404 Handler for Webmasters 

Help users get where they were going. 



technical expertise or hosting facilities. 

Archive-lt to build and browse the colled 



page as it appears now for 



Capture 

as a trusted citation in the future. 

Only available 'or sites that allow crawlers. 








1 










30 scanning centers in 8 countries 
1 ,000 books per day 









; 


0 






Physical Archive 

1 ,500,000 books + plus films, VHS and LPs 







Loq in / Sign Uc 



SUBJECTS 

AUTHORS ADD A BOOK 

LISTS RECENTLY HELP 



LIBRARY 



Search 



One web page lor every book. 



Show only eBooks More search options 




Books to Read The World’s classic literature at your fingertips. Over 1.000.000 free ebook titles available. 




Books to Borrow 



Here’s a sample of recently returned books from the eBook lending library . 



FeIHUC HE 



Borrow books at openlibrary.org 










2,000,000 items 

Feature films, documentaries, commercials, 
propaganda, stock footage, cartoons 
Archive 65 channels of TV 24 hours per day 







TVNews - 



OpenLibrary About 



HU INTERNET ARCHIVE 



Web 



Vtcteo Texts Audio Software 



Account » 



Upload 



TV NEWS ARCHIVE Ukraine 



C Search inside show 



Guide Me 





News 



In-depth coverage of the 
top international stories. 



&OQp/n 




that's why the show 
will be talking about 
the indian elections 
and economy in the 
days and weeks to 
come, that is our 
show for today. I’m ali 
velshi. thanks for 
joining us. good 
evening, everyone, 
this is al jazeera 
america. I'm john 
siegenthaler in new 



&Ofp*n 




crisis, and how 
swapping blood of the 
young with the old 
may reverse the aging 
crisis, and cinco de 
mayo the true story 
behind the unusual 
holiday. J* we begin in 
nigeria where the 
anguish has 
intensified for the 
families of 276 girls 
kidnaps three weeks 



8. -02pm 




about 300 young women 
ages 16 to 18 were told 
they would be taken 
somewhere safe by men 
dressed in police uniforms, 
but the men were actually 
members of boko haram. 
we know little about the 
girls, only some of their 



back our daughters, a 
campaign that began 
on twitter and quickly 
spread, some 
protesters in nigeria 
say they are being 
targeted by the 
government, yvonne 
ndege says 
accusations could stir 
up more anger. » 
reporter these 
allegations could lead 



May 5, 2014 

8.-00 - 9X)1pm EDT 



© Sorrow a DVD 
of this stow 



GRIP 



SCAN 




everything possible to 
support nigeria and 
return the young pern 
to their homes. » 
how they disappeared 
without anybody 
seeing anything has 
been the question all 
along, nigeria has 
been battling boko 
haram for years now. 
the group's name 
means western 



group. » boko haram 
is a frig movement, it 
is a terrorist group 
which has 

assassinated muslim 
learned men, 
terrorized 

communities, and it is 
a small group 
relatively speaking 
but casts large 
shadow. » reporter 
its latest move 



ra 



cinfi il 



n A 






Audio 

• 2,500,000 items 

• 150,000 live concerts from 6,000 bands 

• 60,000 netlabel releases from 2,000 labels 

• Mirrors of Jamendo, IUMA 

• Audio books, radio shows, podcasts